Skip to content

Configure the audio session before recording starts, not only on connect - #1179

Draft
MaxHeimbrock wants to merge 1 commit into
mainfrom
max/fix-ios-audio-session-before-recording
Draft

Configure the audio session before recording starts, not only on connect#1179
MaxHeimbrock wants to merge 1 commit into
mainfrom
max/fix-ios-audio-session-before-recording

Conversation

@MaxHeimbrock

@MaxHeimbrock MaxHeimbrock commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

On iOS the audio engine refuses to open the microphone unless the audio session already permits recording, and livekit_client owns that session: it disables flutter_webrtc's own session management at plugin registration (LiveKitPlugin.swift, setAudioSessionManagementEnabled(false)). But the policy was only pushed to native from Room.connectNativeAudioManagement.start()AudioManager.applyOptionsForConnect()configureNativeAudio — so any recording that starts earlier ran against the app-default soloAmbient category. The engine's pre-enable check rejects that with kAudioEngineErrorAudioSessionInvalidCategory (-9001), surfacing as AudioProcessingException(applyFailed): Audio engine returned error code: -9001.

That is the default path for agent sessions: SessionOptions.preConnectAudio defaults to true, and withPreConnectAudio buffers the microphone before connect. Because startRecording throws before connect is reached, the native policy cache is never seeded, so it fails on every attempt rather than only the first. A pre-join microphone preview hits the same wall with no pre-connect audio involved, which is how #1165 was reported.

The fix pushes the policy from the capture path instead, where the microphone actually opens: LocalAudioTrack.startCapture calls the new AudioManager.prepareRecording() just before Native.startLocalRecording. That one choke point covers pre-connect buffering, a standalone pre-join track, and normal mic publish.

prepareRecording() is deliberately narrow. It is a no-op off iOS, a no-op in AudioSessionManagementMode.manual (the app owns the session and is responsible for a recording-capable category), and a no-op once a policy has been pushed — tracked by a new _hasPushedAppleSessionPolicy flag set at both existing Apple push sites. That last guard is not just an optimization: re-pushing mid-call while the engine has playout only would resolve the playback category through selectCategoryByEngineState and apply it moments before recording starts, so the guard keeps this off the hot path entirely.

The added unit test only asserts the off-iOS no-op — lkPlatform() reads dart:io Platform with no override hook, so no test in this repo can reach an iOS-gated branch. flutter analyze, flutter test, dart format --set-exit-if-changed and import_sorter --exit-if-changed are all clean.

How this went unnoticed

The precondition was never explicit. PreConnectAudioBuffer (#830) opens the microphone outside the room lifecycle by design, and nothing stated who guarantees the session permits recording — it was satisfied incidentally, by flutter_webrtc's ensureAudioSession on getUserMedia and by track-counting that fired on publish, which a pre-connect track has not reached.

Two commits in 2.9.0, a day apart, removed that cover and then made the failure loud:

  • feat(audio): add AudioManager session and routing APIs #1108 (55af814) took sole ownership of the iOS session: it is where setAudioSessionManagementEnabled(false) first appears, killing flutter_webrtc's incidental configuration, and where track-counting became engine-driven lifecycle whose only Dart entry point is connect. After it, no path configures the session before connect.
  • Apply audio processing options when local capture starts #1115 (d1fe342) moved the ADM start into LocalAudioTrack.startCapture and made it fail-fast. Its description mentions the preconnect path, but the review lens there was audio processing options, so the session category beside it went unexamined. Combined with the fork's specific error codes, a latent misconfiguration became a named hard failure.

The fail-fast was not the mistake — it exposed the real one. #1042 reports what looks like the same path failing at 2.7.0 with the generic adm api failed with code: -1, five months earlier, which is likely the same root cause under a code too vague to act on.

Two things kept it out of test range:

  • CI's iOS job is flutter build ios --release --no-codesign — a compile, with no simulator or device run.
  • Unit tests cannot reach the code at all: lkPlatform() reads dart:io Platform with no seam, so on the test host every iOS-gated branch in AudioManager is unreachable. The existing audio tests cover policy resolution and channel argument encoding — pure Dart either side of the platform gate, never the gate itself.

Worth considering separately from this PR: the native pre-enable check is gated #if !TARGET_OS_OSX, so it runs on the simulator too — driving the example app through connect-with-mic there would have caught this end to end, and is the only layer that would have. A test seam for lkPlatform() would make these branches assertable at all, and willEnableEngine finding cachedConfiguration nil is worth logging loudly rather than silently returning proceed and letting the native check fail a layer later.

Fixes #1165

Testing

I tested this locally with the Flutter Agent Starter:

On iOS the audio engine refuses to open the microphone unless the audio
session already permits recording, and livekit_client owns that session:
it disables flutter_webrtc's own session management at plugin
registration. The policy was only pushed to native from Room.connect, so
any recording that starts earlier ran against the app-default
soloAmbient category and was rejected with
kAudioEngineErrorAudioSessionInvalidCategory (-9001), surfacing as
`AudioProcessingException(applyFailed): Audio engine returned error
code: -9001`.

That is the default path for agent sessions, since SessionOptions
enables pre-connect audio, which buffers the microphone before connect
and therefore failed on every attempt. A pre-join microphone preview
hits the same wall without pre-connect audio involved (#1165).

Push the policy from the capture path instead, where the microphone
actually opens. AudioManager.prepareRecording() is a no-op once a policy
has been pushed, so it never re-applies a configuration to a live
session, and in manual mode, where the app owns the session.

Fixes #1165

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hiroshihorie added a commit that referenced this pull request Sep 1, 2026
…pushed (#1182)

## Problem

On iOS the WebRTC audio engine refuses to enable recording unless the
audio session category permits input. Its pre-enable check returns
`kAudioEngineErrorAudioSessionInvalidCategory` (-9001), which the SDK
reported as `AudioProcessingException(applyFailed): Audio engine
returned error code: -9001`.

`livekit_client` owns the iOS audio session since #1108
(`LiveKitPlugin.swift` disables flutter_webrtc's session management at
registration). The native engine observer
(`LKAudioEngineObserver.willEnableEngine`) is the right hook and runs
before the engine's check, but it only applied a configuration that Dart
had pushed. In automatic mode the only push site was `Room.connect`, so
anything that started recording earlier met an empty cache, the observer
returned "proceed" with the session still `soloAmbient`, and the engine
rolled back:

- `Room.withPreConnectAudio`, which `SessionOptions.preConnectAudio`
enables by default, so the Flutter agent starter failed on every "Start
call"
- a pre-join microphone preview (#1165)
- an engine start driven from native before the Flutter side exists, for
example the plugin's static `setEngineAvailability` on a CallKit
killed-state wake

Because the preconnect throw happens before `connect`, the cache was
never seeded and the failure repeated on every attempt.

## How the Swift SDK handles this

`AudioSessionEngineObserver.engineWillEnable` derives the session
configuration from the requested engine state alone (`playAndRecord`
presets while recording, `playback` for playout only), synchronously
inside the engine's enable call. Nothing is configured "before connect".
The engine asks, the observer configures, the engine starts. The Flutter
plugin already has the same observer in the same place, it just had no
built-in policy.

## Fix

`LKAudioEngineObserver.effectiveConfigurationLocked` now resolves a
built-in `playAndRecord` preset (`allowBluetooth | allowBluetoothA2DP |
allowAirPlay`, `videoChat`) whenever nothing has been pushed and
automatic management is on. The existing playout-only `playback` branch
applies to it as well. Manual mode still leaves the session alone. The
Dart-pushed policy becomes an override rather than a prerequisite, and
for the default `AudioSessionOptions.communication` it pushes the same
values, so the connect-time push does not change the live session.

The preset is built on a copy of the shared
`RTCAudioSessionConfiguration.webRTC()` object, and it is best-effort:
if applying it fails, the engine start proceeds and the ADM's own
pre-enable checks still gate recording, so apps whose own session was
already valid are not newly rolled back with -4100. Only a policy the
app actually pushed keeps failing the engine start hard. The mode is
fixed to `videoChat` because it matches the Dart default speaker
preference, and a non-default preference always arrives as a pushed
policy whose mode already carries it, so the preset can never observe
anything else.

`LocalAudioTrack.startCapture` additionally pushes the resolved Dart
policy to native before recording starts (cache-only while the engine is
idle in automatic mode). Flutter-driven starts therefore always use the
real Dart policy, and the built-in preset only stands in for engine
starts that happen before the Flutter side exists, for example the
plugin's static `setEngineAvailability` on a CallKit killed-state wake.

The engine observer is also shared across plugin registrations now: a
second Flutter engine registering in the same process (add-to-app,
`FlutterEngineGroup`) previously reset the pushed policy and management
mode silently, which the preset would have turned into an unwanted
session activation. Only the notification channel is rebound per
registration.

## Error mapping

Audio device module results now get their own error codes on the
`startLocalRecording`, `setEngineAvailability`, `setMicrophoneMuteMode`
and `stopLocalRecording` channels, mirroring client-sdk-swift's
`checkAdmResult`. A mute-mode change can rebuild the engine and hit the
same pre-enable checks as a recording start, so all entry points surface
the same failure the same way:

| ADM result | Native error code | Dart exception |
| --- | --- | --- |
| -9000 `InsufficientDevicePermission` | `deviceAccessDenied` |
`TrackCreateException` |
| -9001 `AudioSessionInvalidCategory` | `audioSessionInvalidCategory` |
`AudioSessionException` (new) |
| -4100 `FailedToConfigureAudioSession` | `audioSessionConfigureFailed`
| `AudioSessionException` (new) |
| anything else | caller fallback (`applyFailed`,
`setEngineAvailability`, ...) | unchanged |

`AudioSessionException` is a new public class with its own changeset
entry.

## Relation to #1179

@MaxHeimbrock's #1179 diagnosed this first and fixes it by pushing the
policy from `LocalAudioTrack.startCapture` (`prepareRecording()`). This
PR ends up covering both layers: `startCapture` pushes the resolved
policy like #1179 does (without a Dart-side flag tracking native cache
state), and the native observer is additionally self-sufficient like
Swift's, so engine enables that never pass through Dart are covered too.
With this merged, `prepareRecording()` becomes redundant.

## Testing

- Flutter agent starter on an iPhone 17 Pro (iOS 27.0): unpatched,
`Start call` failed with -9001 on every attempt. Patched,
`startCapture()` succeeds two seconds before the connect-time policy
push, the preconnect buffer is sent to the agent, and the call connects.
- `flutter analyze`, `flutter test`, `dart format
--set-exit-if-changed`, `import_sorter --exit-if-changed` clean.
- New unit tests cover the error mapping. The iOS-gated branches are not
reachable from unit tests (`lkPlatform()` has no seam), same limitation
as #1179 noted.

Fixes #1165
Refs #1042
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] Audio engine returned error code

1 participant